<!DOCTYPE html>
<html>
<head>
<title>第三章 创建TLS证书和秘钥</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
/* GitHub stylesheet for MarkdownPad (http://markdownpad.com) */
/* Author: Nicolas Hery - http://nicolashery.com */
/* Version: b13fe65ca28d2e568c6ed5d7f06581183df8f2ff */
/* Source: https://github.com/nicolahery/markdownpad-github */

/* RESET
=============================================================================*/

html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video {
  margin: 0;
  padding: 0;
  border: 0;
}

/* BODY
=============================================================================*/

body {
  font-family: Helvetica, arial, freesans, clean, sans-serif;
  font-size: 14px;
  line-height: 1.6;
  color: #333;
  background-color: #fff;
  padding: 20px;
  max-width: 960px;
  margin: 0 auto;
}

body>*:first-child {
  margin-top: 0 !important;
}

body>*:last-child {
  margin-bottom: 0 !important;
}

/* BLOCKS
=============================================================================*/

p, blockquote, ul, ol, dl, table, pre {
  margin: 15px 0;
}

/* HEADERS
=============================================================================*/

h1, h2, h3, h4, h5, h6 {
  margin: 20px 0 10px;
  padding: 0;
  font-weight: bold;
  -webkit-font-smoothing: antialiased;
}

h1 tt, h1 code, h2 tt, h2 code, h3 tt, h3 code, h4 tt, h4 code, h5 tt, h5 code, h6 tt, h6 code {
  font-size: inherit;
}

h1 {
  font-size: 28px;
  color: #000;
}

h2 {
  font-size: 24px;
  border-bottom: 1px solid #ccc;
  color: #000;
}

h3 {
  font-size: 18px;
}

h4 {
  font-size: 16px;
}

h5 {
  font-size: 14px;
}

h6 {
  color: #777;
  font-size: 14px;
}

body>h2:first-child, body>h1:first-child, body>h1:first-child+h2, body>h3:first-child, body>h4:first-child, body>h5:first-child, body>h6:first-child {
  margin-top: 0;
  padding-top: 0;
}

a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 {
  margin-top: 0;
  padding-top: 0;
}

h1+p, h2+p, h3+p, h4+p, h5+p, h6+p {
  margin-top: 10px;
}

/* LINKS
=============================================================================*/

a {
  color: #4183C4;
  text-decoration: none;
}

a:hover {
  text-decoration: underline;
}

/* LISTS
=============================================================================*/

ul, ol {
  padding-left: 30px;
}

ul li > :first-child, 
ol li > :first-child, 
ul li ul:first-of-type, 
ol li ol:first-of-type, 
ul li ol:first-of-type, 
ol li ul:first-of-type {
  margin-top: 0px;
}

ul ul, ul ol, ol ol, ol ul {
  margin-bottom: 0;
}

dl {
  padding: 0;
}

dl dt {
  font-size: 14px;
  font-weight: bold;
  font-style: italic;
  padding: 0;
  margin: 15px 0 5px;
}

dl dt:first-child {
  padding: 0;
}

dl dt>:first-child {
  margin-top: 0px;
}

dl dt>:last-child {
  margin-bottom: 0px;
}

dl dd {
  margin: 0 0 15px;
  padding: 0 15px;
}

dl dd>:first-child {
  margin-top: 0px;
}

dl dd>:last-child {
  margin-bottom: 0px;
}

/* CODE
=============================================================================*/

pre, code, tt {
  font-size: 12px;
  font-family: Consolas, "Liberation Mono", Courier, monospace;
}

code, tt {
  margin: 0 0px;
  padding: 0px 0px;
  white-space: nowrap;
  border: 1px solid #eaeaea;
  background-color: #f8f8f8;
  border-radius: 3px;
}

pre>code {
  margin: 0;
  padding: 0;
  white-space: pre;
  border: none;
  background: transparent;
}

pre {
  background-color: #f8f8f8;
  border: 1px solid #ccc;
  font-size: 13px;
  line-height: 19px;
  overflow: auto;
  padding: 6px 10px;
  border-radius: 3px;
}

pre code, pre tt {
  background-color: transparent;
  border: none;
}

kbd {
    -moz-border-bottom-colors: none;
    -moz-border-left-colors: none;
    -moz-border-right-colors: none;
    -moz-border-top-colors: none;
    background-color: #DDDDDD;
    background-image: linear-gradient(#F1F1F1, #DDDDDD);
    background-repeat: repeat-x;
    border-color: #DDDDDD #CCCCCC #CCCCCC #DDDDDD;
    border-image: none;
    border-radius: 2px 2px 2px 2px;
    border-style: solid;
    border-width: 1px;
    font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
    line-height: 10px;
    padding: 1px 4px;
}

/* QUOTES
=============================================================================*/

blockquote {
  border-left: 4px solid #DDD;
  padding: 0 15px;
  color: #777;
}

blockquote>:first-child {
  margin-top: 0px;
}

blockquote>:last-child {
  margin-bottom: 0px;
}

/* HORIZONTAL RULES
=============================================================================*/

hr {
  clear: both;
  margin: 15px 0;
  height: 0px;
  overflow: hidden;
  border: none;
  background: transparent;
  border-bottom: 4px solid #ddd;
  padding: 0;
}

/* TABLES
=============================================================================*/

table th {
  font-weight: bold;
}

table th, table td {
  border: 1px solid #ccc;
  padding: 6px 13px;
}

table tr {
  border-top: 1px solid #ccc;
  background-color: #fff;
}

table tr:nth-child(2n) {
  background-color: #f8f8f8;
}

/* IMAGES
=============================================================================*/

img {
  max-width: 100%
}
</style>
</head>
<body>
<h1>创建TLS证书和秘钥</h1>
<h2>前言</h2>
<h3>执行下列步骤前建议你先阅读以下内容：</h3>
<ul>
<li><a href="https://rootsongjc.gitbooks.io/kubernetes-handbook/content/guide/managing-tls-in-a-cluster.html">管理集群中的TLS</a>：教您如何创建TLS证书</li>
<li><a href="https://rootsongjc.gitbooks.io/kubernetes-handbook/content/guide/kubelet-authentication-authorization.html">kubelet的认证授权</a>：向您描述如何通过认证授权来访问 kubelet 的 HTTPS 端点</li>
<li>TLS <a href="https://rootsongjc.gitbooks.io/kubernetes-handbook/content/guide/tls-bootstrapping.html">bootstrap</a>：介绍如何为 kubelet 设置 TLS 客户端证书引导（bootstrap）。</li>
</ul>
<blockquote>
<p>注意：这一步是在安装配置kubernetes的所有步骤中最容易出错也最难于排查问题的一步，而这却刚好是第一步，万事开头难，不要因为这点困难就望而却步。</p>
</blockquote>
<h3>kubernetes 系统的各组件需要使用 TLS 证书对通信进行加密，本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 和其它证书；</h3>
<h3>生成的 CA 证书和秘钥文件如下：</h3>
<ul>
<li>ca-key.pem</li>
<li>ca.pem</li>
<li>kubernetes-key.pem</li>
<li>kubernetes.pem</li>
<li>kube-proxy.pem</li>
<li>kube-proxy-key.pem</li>
<li>admin.pem</li>
<li>admin-key.pem</li>
</ul>
<h3>使用证书的组件如下：</h3>
<ul>
<li>etcd：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；</li>
<li>kube-apiserver：使用 ca.pem、kubernetes-key.pem、kubernetes.pem；</li>
<li>kubelet：使用 ca.pem；</li>
<li>kube-proxy：使用 ca.pem、kube-proxy-key.pem、kube-proxy.pem；</li>
<li>kubectl：使用 ca.pem、admin-key.pem、admin.pem；</li>
</ul>
<blockquote>
<p>kube-controller、kube-scheduler 当前需要和 kube-apiserver 部署在同一台机器上且使用非安全端口通信，故不需要证书。 </p>
</blockquote>
<h4>注意：以下操作都在 master 节点即 172.20.200.100 这台主机上执行，证书只需要创建一次即可，以后在向集群中添加新节点时只要将 /etc/kubernetes/ 目录下的证书拷贝到新节点上即可。</h4>
<h1>安装 CFSSL</h1>
<h3>直接使用二进制源码包安装</h3>
<pre><code>wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /root/local/bin/cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /root/local/bin/cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /root/local/bin/cfssl-certinfo

export PATH=/root/local/bin:$PATH
</code></pre>

<p>在$GOPATH/bin目录下得到以cfssl开头的几个命令。</p>
<blockquote>
<p>注意：以下文章中出现的cat的文件名如果不存在需要手工创建。 </p>
</blockquote>
<h3>创建 CA (Certificate Authority)</h3>
<h4>创建 CA 配置文件</h4>
<pre><code>mkdir /root/ssl
cd /root/ssl
cfssl print-defaults config &gt; config.json
cfssl print-defaults csr &gt; csr.json
# 根据config.json文件的格式创建如下的ca-config.json文件
# 过期时间设置成了 87600h
cat &gt; ca-config.json &lt;&lt;EOF
{
  &quot;signing&quot;: {
    &quot;default&quot;: {
      &quot;expiry&quot;: &quot;87600h&quot;
    },
    &quot;profiles&quot;: {
      &quot;kubernetes&quot;: {
        &quot;usages&quot;: [
            &quot;signing&quot;,
            &quot;key encipherment&quot;,
            &quot;server auth&quot;,
            &quot;client auth&quot;
        ],
        &quot;expiry&quot;: &quot;87600h&quot;
      }
    }
  }
}
EOF
</code></pre>

<p>字段说明
- ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
- signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
- server auth：表示client可以用该 CA 对server提供的证书进行验证；
- client auth：表示server可以用该CA对client提供的证书进行验证；</p>
<h3>创建 CA 证书签名请求</h3>
<h4>创建 ca-csr.json 文件，内容如下：</h4>
<pre><code>cat &gt; ca-csr.json &lt;&lt;EOF
    {
      &quot;CN&quot;: &quot;kubernetes&quot;,
      &quot;key&quot;: {
        &quot;algo&quot;: &quot;rsa&quot;,
        &quot;size&quot;: 2048
      },
      &quot;names&quot;: [
        {
          &quot;C&quot;: &quot;CN&quot;,
          &quot;ST&quot;: &quot;BeiJing&quot;,
          &quot;L&quot;: &quot;BeiJing&quot;,
          &quot;O&quot;: &quot;k8s&quot;,
          &quot;OU&quot;: &quot;System&quot;
        }
      ]
    }
EOF    
</code></pre>

<ul>
<li>&quot;CN&quot;：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；</li>
<li>&quot;O&quot;：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；    
</li>
</ul>
<h4>生成 CA 证书和私钥</h4>
<pre><code>$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
</code></pre>

<h4>创建 kubernetes 证书</h4>
<p>创建 kubernetes 证书签名请求文件 kubernetes-csr.json:</p>
<pre><code>cat &gt; kubernetes-csr.json &lt;&lt; EOF
{
    &quot;CN&quot;: &quot;kubernetes&quot;,
    &quot;hosts&quot;: [
      &quot;127.0.0.1&quot;,
      &quot;172.16.200.100&quot;,
      &quot;172.16.200.101&quot;,
      &quot;172.16.200.102&quot;,
      &quot;10.254.0.1&quot;,
      &quot;kubernetes&quot;,
      &quot;kubernetes.default&quot;,
      &quot;kubernetes.default.svc&quot;,
      &quot;kubernetes.default.svc.cluster&quot;,
      &quot;kubernetes.default.svc.cluster.local&quot;
    ],
    &quot;key&quot;: {
        &quot;algo&quot;: &quot;rsa&quot;,
        &quot;size&quot;: 2048
    },
    &quot;names&quot;: [
        {
            &quot;C&quot;: &quot;CN&quot;,
            &quot;ST&quot;: &quot;BeiJing&quot;,
            &quot;L&quot;: &quot;BeiJing&quot;,
            &quot;O&quot;: &quot;k8s&quot;,
            &quot;OU&quot;: &quot;System&quot;
        }
    ]
}
EOF
</code></pre>

<ul>
<li>如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表，由于该证书后续被 etcd 集群和 kubernetes master 集群使用，所以上面分别指定了 etcd 集群、kubernetes master 集群的主机 IP 和 kubernetes 服务的服务 IP（一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.254.0.1。</li>
</ul>
<h4>生成 kubernetes 证书和私钥</h4>
<pre><code>$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
$ ls kubernetes*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem
</code></pre>

<h3>创建 admin 证书</h3>
<p>创建 admin 证书签名请求文件 admin-csr.json：    
</p>
<pre><code>cat &gt; admin-csr.json &lt;&lt; EOF
{
  &quot;CN&quot;: &quot;admin&quot;,
  &quot;hosts&quot;: [],
  &quot;key&quot;: {
    &quot;algo&quot;: &quot;rsa&quot;,
    &quot;size&quot;: 2048
  },
  &quot;names&quot;: [
    {
      &quot;C&quot;: &quot;CN&quot;,
      &quot;ST&quot;: &quot;BeiJing&quot;,
      &quot;L&quot;: &quot;BeiJing&quot;,
      &quot;O&quot;: &quot;system:masters&quot;,
      &quot;OU&quot;: &quot;System&quot;
    }
  ]
}
EOF
</code></pre>

<ul>
<li>后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；</li>
<li>kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；</li>
<li>OU 指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限 </li>
</ul>
<h3>生成 admin 证书和私钥</h3>
<pre><code> $ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
$ ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem
</code></pre>

<h3>创建 kube-proxy 证书</h3>
<p>创建 kube-proxy 证书签名请求文件 kube-proxy-csr.json</p>
<pre><code>cat &gt; kube-proxy-csr.json &lt;&lt; EOF
{
  &quot;CN&quot;: &quot;system:kube-proxy&quot;,
  &quot;hosts&quot;: [],
  &quot;key&quot;: {
    &quot;algo&quot;: &quot;rsa&quot;,
    &quot;size&quot;: 2048
  },
  &quot;names&quot;: [
    {
      &quot;C&quot;: &quot;CN&quot;,
      &quot;ST&quot;: &quot;BeiJing&quot;,
      &quot;L&quot;: &quot;BeiJing&quot;,
      &quot;O&quot;: &quot;k8s&quot;,
      &quot;OU&quot;: &quot;System&quot;
    }
  ]
}
EOF
</code></pre>

<ul>
<li>CN 指定该证书的 User 为 system:kube-proxy；</li>
<li>kube-apiserver 预定义的 RoleBinding cluster-admin 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；</li>
</ul>
<p>生成 kube-proxy 客户端证书和私钥</p>
<pre><code>$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem
</code></pre>

<h3>校验证书</h3>
<p>以 kubernetes 证书为例</p>
<ul>
<li>
<p>使用 opsnssl 命令</p>
<p>$ openssl x509  -noout -text -in  kubernetes.pem
...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=Kubernetes
        Validity
            Not Before: Apr  5 05:36:00 2017 GMT
            Not After : Apr  5 05:36:00 2018 GMT
        Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes
...
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                DD:52:04:43:10:13:A9:29:24:17:3A:0E:D7:14:DB:36:F8:6C:E0:E0
            X509v3 Authority Key Identifier:
                keyid:44:04:3B:60:BD:69:78:14:68:AF:A0:41:13:F6:17:07:13:63:58:CD</p>
<pre><code>        X509v3 Subject Alternative Name:
            DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:172.20.0.112, IP Address:172.20.0.113, IP Address:172.20.0.114, IP Address:172.20.0.115, IP Address:10.254.0.1
</code></pre>

<p>...</p>
</li>
<li>
<p>使用 cfssl-certinfo 命令    
</p>
<p>$ cfssl-certinfo -cert kubernetes.pem
...
{
  &quot;subject&quot;: {
    &quot;common<em>name&quot;: &quot;kubernetes&quot;,
    &quot;country&quot;: &quot;CN&quot;,
    &quot;organization&quot;: &quot;k8s&quot;,
    &quot;organizational</em>unit&quot;: &quot;System&quot;,
    &quot;locality&quot;: &quot;BeiJing&quot;,
    &quot;province&quot;: &quot;BeiJing&quot;,
    &quot;names&quot;: [
      &quot;CN&quot;,
      &quot;BeiJing&quot;,
      &quot;BeiJing&quot;,
      &quot;k8s&quot;,
      &quot;System&quot;,
      &quot;kubernetes&quot;
    ]
  },
  &quot;issuer&quot;: {
    &quot;common<em>name&quot;: &quot;Kubernetes&quot;,
    &quot;country&quot;: &quot;CN&quot;,
    &quot;organization&quot;: &quot;k8s&quot;,
    &quot;organizational<em>unit&quot;: &quot;System&quot;,
    &quot;locality&quot;: &quot;BeiJing&quot;,
    &quot;province&quot;: &quot;BeiJing&quot;,
    &quot;names&quot;: [
      &quot;CN&quot;,
      &quot;BeiJing&quot;,
      &quot;BeiJing&quot;,
      &quot;k8s&quot;,
      &quot;System&quot;,
      &quot;Kubernetes&quot;
    ]
  },
  &quot;serial</em>number&quot;: &quot;174360492872423263473151971632292895707129022309&quot;,
  &quot;sans&quot;: [
    &quot;kubernetes&quot;,
    &quot;kubernetes.default&quot;,
    &quot;kubernetes.default.svc&quot;,
    &quot;kubernetes.default.svc.cluster&quot;,
    &quot;kubernetes.default.svc.cluster.local&quot;,
    &quot;127.0.0.1&quot;,
    &quot;10.64.3.7&quot;,
    &quot;10.254.0.1&quot;
  ],
  &quot;not</em>before&quot;: &quot;2017-04-05T05:36:00Z&quot;,
  &quot;not_after&quot;: &quot;2018-04-05T05:36:00Z&quot;,
  &quot;sigalg&quot;: &quot;SHA256WithRSA&quot;,
...</p>
</li>
</ul>
<h3>分发证书</h3>
<p>将生成的证书和秘钥文件（后缀名为.pem）拷贝到所有机器的 /etc/kubernetes/ssl 目录下备用；</p>
<p>mkdir -p /etc/kubernetes/ssl
cp *.pem /etc/kubernetes/ssl</p>

</body>
</html>
<!-- This document was created with MarkdownPad, the Markdown editor for Windows (http://markdownpad.com) -->
